Semantic Graph Visualisation for Mobile Semantic Web Interfaces

Information visualisation benefits from the Semantic Web: multimodal mobile interfaces to the Semantic Web offer access to complex knowledge and information structures. Natural language dialogue systems are ideal interfaces to personal digital assistants (PDAs) or other handheld clients. We explore more fine-grained co-ordination of multimodal presentations as answers to natural language questions about a specific domain by graph-based visualisation and navigation in ontological RDF result structures. Semantic Navigation on mobile devices leverages graphical user interface activity for dialogical interaction in mobile environments. Constraint-based programming helps to find optimised multimedia graph visualisations. Introduction For every specific type of information there are certain categories of visual representations that are more suitable than others. The use of a graph for the visualisation of information has the advantage that it can capture a detailed knowledge structure. Therefore graphs are suitable for conveying semantic relations between individual information items and for providing an understanding of the overall information structure. Apart from that dialogue systems are very useful for interacting with Web-based information services in mobile environments. The challenge we address is the intuitive navigation in a structured, semantically organised information space on small interaction devices such as PDAs. Our aim is to implement and evaluate mobile Semantic Web interfaces by application of direct structure mapping from RDF graphs toward their multimedia visualisations. Displaying RDF data in a user-friendly manner is a problem addressed by various types of applications using different representation paradigms [1]. At least the following types can be identified: keyword search, e.g. Swoogle, faceted browsing [2], explicit queries, e.g. Sesame, and graph visualisations. In our approach we use an automatic layouter for dynamic constrained graph display tailored to suit small displays as valuable extension to other graph visualisation techniques. By additional graph presentation capabilities in a primary 1 http://www.w3.org/RDF/ 2 http://swoogle.umbc.edu/ 3 http://www.openrdf.org/ Fig. 1. Graphical user interface and semantic navigation (centre and right). linguistic question answering scenario, the users would become more engaged in the dialogue, navigate through the incrementally presented result space, and would be encouraged to pose follow-up questions in natural language. In our most recent dialogue system project SmartWeb [3] following experiences in [4– 6], we try to provide intuitive multimodal access to a rich selection of Web-based information services; especially the handheld scenario is tailored toward multimodal interaction with ontological knowledge bases and Semantic Web services [7] in the football domain. For example, the user can ask questions like How many goals has Michael Ballack scored this year? or How did Germany play against Argentina in the FIFA World Cup? The summarised answer to the last question, SmartWeb provides and synthesises, is 5 Spiele (5 matches). Figure 1 shows the PDA interaction device. When the structure of the dynamic graph changes, a new optimal layout is computed server-side. A further click on the Ergebnis (result) node results in displaying the information: 5:3 n. E., 1:1 n. V. (1:1, 0:0), Ereignis (incidence) reveals red card for player Cufre, for example. A shared representation and a common ontological knowledge base ease the data flow among components and avoid costly transformation processes [8]. This applies to the visualisation process, too. We use the ontological RDF metadata to arrange information pieces in automatically layout graphs with respect to their semantic relations extracted from RDF results obtained from our knowledge servers. Humans themselves may encode information based upon its meaning [9]. Users feel familiar with this way of information arrangement at least. Integration We integrated the semantic graph visualisation approach into our distributed dialogue system. In SmartWeb, the graph-based user interface on the client is connected to the automatic graph layouter that resists on the server. All data transfer between server and client is organised by special XML structures transmitted over socket connections in both directions. We extended this XML structure by an new dynamic graph environment, for the graph structure data to be exchanged, the graph node layout positions, and the user interactions. The data flow is shown in figure 2. The presentation capabilities after integration